Getting Updates
On API is a set of streaming endpoints designed for staying up to date with the latest data. Gracenote assigns each customer their own data stream containing only the data pertinent to their subscription, aka API “entitlement”. Whenever a change is made on any entitled object in a given endpoint, an incrementing updateId number is assigned to that latest version of the object, and the entire object is moved to the “tail” of the endpoint (endpoint order is objects with oldest changes first).
You can keep your data up to date by periodically querying for updates that have occurred since the last ingested updateId. To load/reload endpoint data in its entirety, start with updateId=0.
API
http://on-api.gracenote.com/v3/[endpoint]
?updateId=[updateId value] &limit=[limit value] &api_key=[your-api-key]
Request Parameters
| Parameter | Description |
|---|---|
| updateId | ID token used to make incremental requests. |
| limit | Number of records to be requested. 1000 is the standard and maximum allowed. |
| api_key | Your API key |
The On API endpoint response includes a nextUpdateId field and a maxUpdateId field within the streamData node. Use the nextUpdateId value to continue fetching the endpoint data; Once you reach the current end of updates (EOS, “end of stream”), the nextUpdateId field will not be provided. The maxUpdateId value is always provided for calls using this delivery mechanism. Once you reach the EOS, the next updateId to request will be maxUpdateId + 1.
Important: Due to the large volume of data entitlements, Gracenote engineering has enabled a limit parameter to control batch size of each request.
We recommend requesting with a limit of 1000 records per query across all endpoints except Lineups. The maximum allowable limit for the Lineups endpoint is 10.Higher limit values are not supported and are hard limited to the above mentioned maximums (i.e. will not result in an error but the API will ignore the requested limit parameter if over the maximum).
We do our best to optimize the API for all of our global customers. However, if you encounter performance issues like the response taking too long, or not getting a response at all, we recommend retrying with a reduced batch size.
Data Structure
The On API is designed to deliver a stream of updates for any given data set. This returned XML consists of the Header containing requestParameters and streamData nodes, and the payload containing the list of actual endpoint objects such as Programs. The Header requestParameters reflect the parameters from the URL; this can be useful for validation/troubleshooting purposes. The streamData node contains the nextUpdateId and maxUpdateId values. These two values are also returned in the HTTP response header. We recommend using the ones from the XML response.
<on xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ... >
<script/>
<header>
<content>On - Updates: Sources</content>
<created>2024-08-23T00:38:44Z</created>
<copyright>Copyright 2024 Gracenote, a Nielsen Company. ... </copyright>
<requestParameters>
<requestParameter name="updateId">0</requestParameter>
<requestParameter name="limit">10</requestParameter>
</requestParameters>
<streamData>
<nextUpdateId>4250128775</nextUpdateId>
<maxUpdateId>6722796218</maxUpdateId>
</streamData>
</header>
<sources>
...
</sources>
</on>
Update Process
For each endpoint:
- For the seeding/“cold start”, begin fetching data by requesting with parameter updateId=0.
- Iterate through all updates by using the nextUpdateId value as the request parameter for the next call. The last iteration (end of stream) is indicated by not returning any nextUpdateId value.
- Having reached the end of the data stream, go idle and start requesting updates (step #2) sometime later using the stored maxUpdateId +1 value. See “Data Update Frequency” below for guidance on the idle time.
- If API/download errors are encountered, wait and retry up to N times, then exit if error persists; note the nextUpdateId from the last successfully fetched response as the updateId from which to resume fetching - assuming responses prior to encountering the error will have been processed/ingested.
The top-level objects in each endpoint carry identifier field(s) allowing to uniquely identify them, along with the updateId and updateDate. The objects are provided in their entirety and should replace older existing objects (with lesser updateId) in the customer database / store. It is recommended to store updateId and updateDate along with the objects.
There may be multiple instances (different updateIds) of the same object (same identifier) in a single download session - the instance with the largest/latest updateId should be used. There should not be object duplicates (same identifier, same updateId) in a single download session, but if this may cause an issue downstream (e.g. attempt to insert duplicate key), we recommend validating against such a scenario.
For object deletion / inactivation, please see Removals (Object Deletion / Inactivation)
For updates to Imagery, please see Managing Image Updates and Deletes
Endpoint Update Sequence
Endpoints can be retrieved sequentially in any order, though because they update asynchronously, it may be beneficial to e.g. retrieve Programs prior to Schedules to maximize (Schedules->Programs) referential integrity.
Endpoints can also be retrieved in parallel. With parallel endpoint retrieval, the On API QPS limits may result in some calls returning HTTP 403 “Over QPS limit” errors. The recommended approach is either to wait and retry (up to N times) on HTTP 403 error, or to implement QPS throttling. Contact your Gracenote representative if you need more details about this scenario.
Sequential retrieval of endpoints is highly unlikely to trigger “Over QPS limit” error, unless the requests set the limit parameter significantly below recommended value which is 1000 for most endpoints.
Gracenote does not support parallel retrieval on a single endpoint using the nextUpdateId value returned in the response header (while payload download is still in progress).
Data Update Frequency
Data is continuously being added to / updated in the Gracenote systems, and made available via the On API. The client sessions retrieve, process and ingest the endpoint data until encountering the end-of-stream. Depending on your use cases, the session interval for your implementation may be smaller (from 15 min to 1 hr) or larger (e.g. 1 day). For most linear schedule/streaming use cases, an hourly session cadence should be sufficient. The new session should start only if the previous session has finished (on the same endpoint).
For information on expected update volume, see the System Requirements section.
Reseeding
For periodic database refresh, restart API ingestion from updateId=0. The On API data stream only maintains the newest (highest updateId) version of an object, so it will not be streaming through every historical update. However, depending on the size of the API stream / entitlement, the data reseeding via the API can be a lengthy process. Reseeding may be necessary if Gracenote enables a new attribute on existing objects in an endpoint (e.g. video descriptors on programs). When done for a specific client stream, this action typically does not trigger object updates on the endpoint.
Removals (Object Deletion / Inactivation)
On API uses two attributes to indicate when updates for a record stop being available in a certain endpoint:
- “deleted=true” (referred to as “delete” below)
- “inactive=true” (referred to as “inactive” below)
<program TMSId=... updateId="77374268383" deleted="true" updateDate="2024-06-18T13:50:30Z"> <program TMSId=... updateId="78878358803" inactive="true" updateDate="2024-07-25T01:39:16Z">
Most endpoints use the deleted attribute only. Exceptions are:
- The Programs endpoint uses both delete and inactive attributes.
- Schedule endpoint provides data in “schedule-days” (one UTC day schedule for a single channel). If a schedule-day is outside of your entitled EPG time window, the On API will automatically stop publishing updates for those days. Schedule endpoint does not use deletes and you can decide within the limits of your contract when to remove the old schedule records from your databases.
- Images provided within Programs, Celebrities, Sources and other endpoints, use a separate set of attributes to signal updates and deletions. See Managing Image Updates and Deletes.
- While not strictly an entity removal, channel positions in Lineups Endpoint endpoint use an expiration date.
Processing Deleted Records
The deleted attribute indicates that the record and its ID have been removed from Gracenote's database, or removed from your entitlement. You will no longer receive updates for the deleted record.
Guidelines
- Generally, remove the record from your database.
- Alternatively, if your use case assumes references to old records, you can keep the record marked as "deleted" in your database.
Ensuring Referential Integrity
- Sometimes, deleted records may still be referenced in other endpoints due to asynchronicity of the API. In these cases, consider postponing removing the record until all other references are updated.
- If references exist in archived data, you may choose to keep deleted records in your database.
Records flagged as deleted are retained in the On API stream for a certain time period, in the same state as at the time of deletion. As a result, these records may still be returned during lookups or reseeding. This supports history and transparency of the changes.
Important: Identifiers are not reused for different objects. However, a record and its same identifier may reappear in certain cases, including after an erroneous deletion, re-entitlement of a streaming catalog, or after remapping of a streaming asset to the same program (for ProgramAvailabilities endpoint).
When a record is flagged as deleted, the body of the XML may or may not display information previously available for that record. The deleted record will persist in the output and will not be purged for a preset number of days (see Days Retained above). In some cases, the XML of “deleted” records may stop conforming to the XSD schema. Therefore, we recommend not validating “deleted” records
Deleted Record Example
<program TMSId="EP000000351352" rootId="13813823" deleted="true" connectorId="SH000000350000" seasonId="13314474" seriesId="184224" updateId="53343177" updateDate="2017-04-21T04:51:13Z" >
<titles lang="en">
<title size="120" type="full" subType="Main" lang="en">60 Minutes</title>
...
</program>
Processing an Inactive Program
The inactive=true indicates that the Program was removed from your entitlement, generally due to not being referenced on a linear schedule or in streaming catalogs. You will no longer receive updates, but the Program can reappear in the endpoint in the future (i.e. become active again with a new updateId).
| Endpoint | inactive="true" | Days Retained |
|---|---|---|
| Programs | Program record is no longer referenced on linear schedules or in streaming catalogs and/or has been removed from your entitlement | 60 days |
Guidelines
Generally, retain the Program but mark it as "inactive". You may choose to remove the record from your database or retain it if your use case references older programs. Programs that are flagged as inactive are retained in the On API stream for 60 days in the same state as when inactivated. As a result, these Programs will still be returned during lookups or reseeding within this period. This supports history and transparency of the changes. Inactive Programs can become active again and resume receiving regular updates.